WORLD INTELLECTUAL PROPERTY ORGANIZATION 
International Bureau 




PCT 

INTERNATIONAL APPLICATION PUBLISHED UNDER THE PATENT COOPERATION TREATY (PCT) 



(51) International Patent Classification 6 x 
G06T 5/00, H04N 1/60 // 9/64 



Al 



(11) International Publication Number: WO 97/01151 

(43) International Publication Date: 9 January 1997 (09.01.97) 



(21) International Application Number: PCT/FI96/00339 

(22) International Filing Date: 7 June 1996 (07.06.96) 



(30) Priority Data: 
953061 



20 June 1995 (20.06.95) 



FI 



(71) Applicant (for all designated States except US): TEKNILLI- 

NEN KORKEAKOULU [FI/FI]; Graafisen tekniikan labo- 
ratories Tekniikantie 3 A, FIN-02150 Espoo (FT). 

(72) Inventors; and 

(75) Inventors/Applicants (for US only): SAARELMA, Hannu 
[FI/FI]; TKK/Graafisen tekniikan laboratorio, Tkeniikantie 
3 A, FIN-02150 Espoo (FI). LAIHANEN, Pekka [FI/FI]; 
TKK/Graafisen tekniikan laboratorio, Tekniikantie 3 A, FIN- 
02150 Espoo (FI). 

(74) Agent: PAPULA REIN LAHTELA OY; P.O. Box 981, 
Frcdrikinkatu 61 A f FIN-00100 Helsinki (FI). 



(81) Designated States: JP, US, European patent (AT, BE, CH, DE, 
DK, ES, FI, FR, GB, GR, IE, IT, LU, MC, NL, PT t SE). 



Published 

With international search report 

Before the expiration of the time limit for amending the 
claims and to be republished in the event of the receipt of 
amendments. 

In English translation (filed in Finnish). 



(54) Title: METHOD AND APPARATUS FOR MAXIMIZING THE VISUAL QUALITY OF IMAGE PRESENTED IN ELECTRIC 
FORM 

(57) Abstract 

The present 
invention relates to 
a procedure and an 
apparatus for optimizing 
the visual quality of an 
image. In the procedure, 
the greyness balance of 
the image is adjusted by 
a statistical method and 
the colour reproduction 
is adjusted to make it 
correspond to a standard 

observer's preference, which is a predetermined optimum value for colour reproduction. The procedure can be used e.g. to improve 
the quality of images transmitted over information networks by providing the image transmission path with an apparatus which corrects 
images detected in the transmission path by applying the procedure of the present invention. 
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METHOD AND APPARATUS FOR MAXIMIZING THE VISUAL QUALITY 
OF IMAGE PRESENTED IN ELECTRIC FORM 

The present invention relates to a procedure 
for maximizing the visual quality of an image as de- 
5 fined in the preamble of claim 1. 

In addition, the present invention relates to 
an apparatus for maximizing the visual quality of an 
image as defined in the preamble of claim 10. 

As the use of information networks is gaining- 

10 ground and their data transmission capacity is increa- 
sing, increasing numbers of digital colour images are 
being transmitted across the networks. Images are sent 
into the network from various sources, using different 
equipment and various encoding and packing methods 

15 known in themselves. Moreover, the technical origin of 
the images is generally unknown to the receiver. 

Due to errors caused by the operations per- 
formed on the image, such as encoding and packing, as 
well as errors generated in the data transfer medium, 

20 the visual quality of the image is often deteriorated 
as compared with its original visual quality. For this 
reason, methods for implementing the colour reproduc- 
tion of an image have been developed to maximize the 
visual quality of the image. In an optimal case, the 

25 reproduced image reproduces the visual impression gi- 
ven by an error-free original image unchanged. In 
practice, however, this is not always possible. A . ba- 
sic reason for this is that the original image usually 
has a larger range of colours than is possible to 

30 reproduce. But even if all the colours in the original 
could be reproduced, creating a visual similarity 
would still be difficult. That is to say, current 
chromatics does not know any methods for quantitative- 
ly determining the colour impressions produced by nor- 

35 mal images consisting of numerous fields of different 
colours and shapes. 

Several methods for colour correction and ca- 



WO 97/01151 



PCT/FI96/00339 



libration, for improving the visual quality of an ima- 
ge, are known in the art. A feature common to these 
methods is that the technical origin of the image is 
known or a "model" (a high-quality photograph) of the 

5 image exists. However , the technical origin of an ima- 
ge received e.g. over an information network is not 
known, and therefore the prior-art methods cannot be 
used to improve the visual quality of the image. 

It is possible to improve the visual quality 

0 of an image by a manual method, whereby an operator 
processing the image makes observations about the ori- 
ginal picture, an image displayed on a computer screen 
or printed out on a printer, and corrects the colour 
reproduction manually on the basis of his/her own ob- 

5 servations and skill. However, processing an image by 
this manual method is a slow and laborious business 
and requires a skilled image processing operator. Mo- 
reover, manual image processing requires expensive ma- 
chines which must be compatible with each other and 

0 calibrated to ensure e.g. that an image displayed on- 
screen visually corresponds to its printed copy. 

The object of the present invention is to 
eliminate the problems and drawbacks described above. 

A specific object of the present invention is 

5 to produce a new type of procedure for image proces- 
sing which can be easily implemented and automated for 
use in the processing of large amounts of images. 

A further object of the present invention is 
to produce an apparatus which can be used for the pro- 

0 cessing of an image being transmitted in an informati- 
on network, a printing system or the like to optimize 
the visual quality of the image without significantly 
retarding its transmission. 

An additional object of the present invention 

5 is to present a procedure which can be used for impro- 
ving the visual quality of an image without special 
image processing skill. 
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As for the features characteristic of the 
present invention, reference is made to the claims. 

In the procedure of the present invention, an 
image in electric format, preferably a colour image, 
5 is processed. The image may be a packed one, obtained 
by packing methods known in themselves, e.g. methods 
employing the GIF or TIFF standard. According to the 
invention, the image is converted into a chromaticity 
coordinate format, preferably an Lsa coordinate for- 

10 mat, where L stands for luminance, s for colour satu- 
ration, and a for hue angle. The Lsa values are de- 
termined for the RGB values in Fig. 1 according to 
predefined equations. The image can also be converted 
into a format employing a different coordinate system. 

15 After this, the greyness balance of the image is de- 
termined and compared with a standard observer's grey- 
ness balance preference. Based on this comparison, the 
greyness balance of the image is adjusted to make it 
correspond to the standard observer's preference. In 

20 connection with the adjustment of the greyness balance 
in a preferred case, the highest and lowest luminance 
values of the grey shades are determined, a probabili- 
ty model is used to determine whether the shades 
should be neutral grey and the whole image is shifted 

25 in the colour space so that the highest and the lowest 
values correspond to neutral grey. The purpose of this 
operation is to adjust the greyness balance of the 
image so that white will be perceived as white and 
black will be perceived as black to a maximum degree 

30 of "correctness". Finally, according to the invention, 
the optimized image is converted from the chromaticity 
coordinate format back into its original format. 

The colour reproduction of the image is opti- 
mized to make it correspond to a standard observer's 

35 preference. A visual processor has been taught a pre- 
ference selected on the basis of a large number of 
images of various themes presented to a number of test 
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persons. The visual processor adjusts the visual qu- 
ality of each image to a maximum level. The teaching 
of the visual processor can be performed by a control 
parameter optimizing method or a neural network met- 
5 hod. 

The procedure of the invention has several 
advantages as compared with prior art. The most impor- 
tant and most significant advantage is undoubtedly the 
fact that the procedure corrects the visual quality of 

10 the image automatically and independently of the sour- 
ce, in other words, the technical origin as well as 
the "correct" colour reproduction are unknown. Furt- 
hermore, the procedure can be implemented independent- 
ly of hardware, which means that the display device or 

15 - printer eventually used to reproduce the image need 
not be known. The procedure can also be provided with 
profiles of different output device groups or indivi- 
dual devices, and the procedure can be incorporated in 
individual output devices. 

20 A further advantage of the present invention 

as compared with prior art is that the procedure can 
be easily standardized for use as a method of impro- 
ving image quality in information networks. Moreover, 
the procedure can be applied for different types of 

25 display devices, including the cathode ray tube, li- 
quid crystal display and electroluminescent display. 
Further, the procedure can be applied for different 
types of display control systems, including standardi- 
zed computer screens and computer screens developed 

30 for special uses, as well as television screens emplo- 
ying the normal and HDTV standards. In addition, the 
procedure can be applied for different types of colour 
printer, including electrography, ink jet, thermal 
printers, electronic colour printing machines and ot- 

35 her printing machines connectable to a computer. 

In an embodiment of the present invention, 
the colour reproduction of an image is determined and 
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compared with a predetermined colour reproduction pre- 
ference of a standard observer and, based on this com- 
parison, the colour reproduction of the image is • ac- 
centuated to make it correspond to the standard obser- 
5 ver's preference. Preferably the colour reproduction 
of the image is accentuated by adjusting the global 
colour reproduction of the image to make it correspond 
to the standard observer's preference. Global colour 
reproduction means the colour reproduction in the en- 

10 tire image, in other words, at this stage of the pro- 
cedure the colours in the entire image are crispened 
or accentuated. 

In another embodiment of the invention, the 
colour reproduction of the image is accentuated by lo- 

15 eating local colour areas on the basis of occurrence 
probabilities and adjusting the colour reproduction in 
the image areas encountered to make it correspond to 
the standard observer's preference. In searching the 
image for local colour areas, detected continuous 

20 areas of sufficient size are compared with predefined 
values. A given standard colour can be defined before- 
hand e.g. for the human skin, the sky, water and any 
objects, such as a Coca Cola can. From the shape, si- 
ze, position and size of chromaticity statistic of a 

25 colour area encountered, it is possible to infer whet- 
her it is a predefined colour type. If such as colour 
type is encountered, the local colour area in the ima- 
ge is adjusted to make it correspond to the predefined 
colour. 

30 In an embodiment of the present invention, 

the image is classified according to certain classifi- 
cation rules. First, the image can be subjected to a 
certain type of thematic classification, to determine 
e.g. whether the image represents a portrait or a 

35 landscape. In addition, the image can be subjected to 
a certain type of quality classification to determine 
the quality of the image. Quality classes have prefe- 
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rably been defined beforehand. The quality classifica- 
tion is used to prevent "over-optimization" of an ima- 
ge, i.e. to ensure that an image whose quality has. al- 
ready been visually optimized by the method of the in- 
5 vention will not be processed again. 

The standard observer's preference mentioned 
in this application is determined by presenting a num- 
ber of images to a large number of people and using 
their observations to determine optimal values for the 

10 greyness balance, global colour reproduction and local 
colour reproduction of the images. Let it be stated 
that the standard observer' s preference can be conti- 
nuously "improved" by increasing the number of people 
to whom the images are presented. On the other hand, 

15 it is also possible to use a standard observer's pre- 
ference produced by a highly skilled image processing 
specialist. This can be achieved e.g. by teaching that 
preference to the apparatus implementing the procedu- 
re* 

20 The apparatus of the present invention for 

optimizing the visual quality of an image in electric 
format comprises a visual processor which stores pre- 
defined standard observer's preference values for 
greyness balance and colour reprodution and which 

25 receives an image, analyzes and classifies it and ma- 
kes decisions as to how the image is to be adjusted, 
and a signal processor electrically connected to the 
visual processor to adjust the image as controlled by 
the visual processor. 

30 The visual processor preferably comprises 

conversion means for converting the image into an Lsa 
coordinate format. Further, the conversion means pre- 
ferably consist of electric components in the proces- 
sor circuit. The signal processor may be any kind of 

35 microprocessor, programmable circuit or application- 
specific integrated circuit (ASIC) suited for use in 
signal processing. 
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As for the advantages of the apparatus of the 
invention, reference is made to those of the procedure - 
of the invention. 

In the following, the invention is described 
5 by the aid of examples of its embodiments by referring 
to the attached drawings, in which 

Fig. 1 presents a block diagram representing 
an implementation of the procedure of the invention; 

Fig. 2 presents a diagram representing an ap- 
10 paratus as provided by the present invention; and 

Fig. 3 represents a chromaticity coordinate 

format. 

Referring to the block diagram in Fig. 1, the 
procedure of the invention works as follows. An image 

15 e.g. in an RGB format is received in block 4 and con- 
verted into an Lsa format. This conversion is descri- 
bed later on by referring to Fig, 3. After this, an 
adjustment of the greyness balance is effected in 
block 5. In block 5, highest and lowest values for pa- 

20 rameter L are determined and compared with predefined 
values. If necessary, based on the comparison, the 
whole Lsa colour space of the image is shifted so that 
the highest and lowest values fall on the desired 
point in the colour space. 

25 After the adjustment of the greyness balance, 

the colour reproduction of the image is accentuated. 
The stages of the accentuation process are represented 
by blocks 6 and 7. In block 6, colour reproduction is 
accentuated globally concerning the whole image, and 

30 in block 7 local colour areas are accentuated. The lo- 
cal colour areas are determined by finding certain 
predefined hues and colours close to them and adjus- 
ting the areas encountered to suitable values. After 
both greyness balance and colour reproduction have 

35 been corrected, an Lsa => RGB conversion is performed 
in block 8 to obtain new R x , G x and B x values. 

The apparatus represented by Fig. 2 comprises 
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in its simplest form a visual processor 1, which clas- 
sifies the image and performs the necessary operations 
to determine how the image is to be corrected. Moreo- 
ver, the apparatus comprises a signal processor 2, 
5 which is used to perform the RGB => Lscc conversion and 
is electrically connected to the visual processor 1. 
Controlled by the visual processor, the signal proces- 
sor makes the required changes in the image. The appa- 
ratus shown in Fig. 2 also comprises a printer or a 
10 display 3 used to print or display the corrected ima- 
ge. 

Referring now to Fig. 3, the RGB => Lsa con- 
version performed in the first stage of the procedure 
of the invention is described. The colour space repre- 
15 sented by Fig. 3 is of a cylindrical shape. The lumi- 
nance L is determined from the RGB values according to 
equation (1) as follows: 

max * plus 

where: V = k x R + k 2 + k 3 B 

V max = V / max(R,G,B) 
v pius = a(l - Vmax), and 
a = user-defined constant 
ki + k 2 + k 3 = 1, and 
kif k 2 , k 3 , R, G, B and a € [0, 1] 
The significance of the R, G and B values for 
the new parameters can be adjusted by means of the 
constants k lf k 2 and k 3 , and the proportion of neutral 
and saturated colours by means of the constant a. 

The colour reproduction angle a 0 < a < 2n is 
calculated from equation (2) as follows: 
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a = arctan 



9 



r 2G-R-B) n 0 
,R>3 



(ig-r-b) n n 

a = arctan . / + n,B > R (2) 

(if R = B;a = 3/2x; if a> Q.a = n + 2n) 

Further, colour saturation s is calculated 
from the RGB values using equation (3) as follows: 

5 

_ max(R 9 G,B)-min(R,G 9 B) 
max(i2,G,5) 

After the required operations have been per- 
formed on the image in the Lsa space, analogies of 

10 equations (1), (2) and (3) can also be used for the 
Lsa ==> RGB conversion to obtain new R lf G x and B 1 va- 
lues. Let it be noted that in this example only one 
chromaticity coordinate conversion has been described 
and that other conversions can also be used within the 

15 scope of the inventive idea. 

The invention is not limited to the embodi- 
ment examples discussed above, but many variations are 
possible within the scope of the inventive idea de- 
fined by the claims. 
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CLAIMS 

1. Procedure for automatically maximizing the 
visual quality of an image in electric format, charac- 
terized in that the following operations are performed 

5 automatically: 

— the image is converted into a chromaticity 
coordinate format 

— a probable greyness balance is determined 
for the image by a statistical method from the video 

10 signal and the greyness balance of the image is adjus- 
ted to correct it; 

— the colour reproduction of the image is ad- 
justed; and 

— the maximized image is converted from the 
15 chromaticity coordinate format back into its original 

format . 

2. Procedure as defined in claim 1, characte- 
rized in that the colour reproduction of the image is 
determined and compared with a certain colour repro- 

20 duction preference of a standard observer and, based 
on this comparison, the colour reproduction of the 
image is accentuated to make it correspond to the 
standard observer's preference. 

3. Procedure as defined in claim 1 or 2, cha- 
25 racterized in that in an Lsa coordinate format, L 

stands for luminance, s for colour saturation, and a 
for the hue angle, these values being determined from 
the RGB values of the image using predefined equa- 
tions . 

30 4. Procedure as defined in any one of the 

preceding claims 1- 3, characterized in that the co- 
lour reproduction of the image is accentuated by ad- 
justing the global colour reproduction of the image to 
make it correspond to the standard observer' s pref e- 

35 rence. 

5. Procedure as defined in any one of the 
preceding claims 1- 4, characterized in that the co- 
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lour reproduction of the image is accentuated by fin- 
ding local colour areas on the basis of occurrence 
probabilities and adjusting the colour reproduction in 
the areas encountered in the image to make it corres- 
5 pond to the standard observer's preference. 

6. Procedure as defined in any .one- of the 
preceding claims 1-5, characterized in that a histo- 
gram is generated to represent the colour space of the 
image, the highest and lowest values of the grey sha- 

10 des are determined, the values are compared with pro- 
bability values and, based on this comparison, the 
image is shifted in the colour space so that the 
highest and lowest values correspond to neutral grey. 

7. Procedure as defined in any one of the 
15 preceding claims 1-6, characterized in that the the- 
me of the image is determined and the image is classi- 
fied according to its theme into a predefined theme 
among a number of predefined themes. 

8. Procedure as defined in any one of the 
20 preceding claims 1-7, characterized in that the qu- 
ality level of the image is determined and the image 
is classified according to its quality level into a 
predefined quality level among a number of predefined 
quality levels. 

25 9. Procedure as defined in any one of the 

preceding claims 1 - 8, characterized in that the 
standard observer' s preference is determined by pre- 
senting a number of images to a large number of people 
and using their observations to determine optimal va- 

30 lues for the greyness balance, global colour reproduc- 
tion and local colour reproduction of the images. 

10. Apparatus for optimizing the visual qu- 
ality of an image in electric format, characterized in 
that the apparatus comprises a visual processor (1) in 

35 which predetermined standard observer's preference va- 
lues for greyness balance and colour reproduction are 
stored and which receives the image, analyzes and 
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classifies it and makes the decisions as to how the 
image is to be adjusted, and a signal processor (2) 
electrically connected to the visual processor to al- 
low adjustment of the image under the control of the 
5 visual processor. 

11. Apparatus as defined in claim 10, charac- 
terized in that the visual processor (1) comprises 
conversion means for converting the image to an Lsa 
coordinate system. 
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